DOCUMENT RESUME 



ED 444 520 



IR 020 176 



AUTHOR 
TITLE 
PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Steed, Mario 

Role of Audience in Multimedia Production. 

2000 - 00-00 

9p.; In: Society for Information Technology & Teacher 
Education International Conference: Proceedings of SITE 2000 
(11th, San Diego, California, February 8-12, 2000) . Volumes 
1-3; see IR 020 112. 

Reports - Research (143 ) - - Speeches/Meeting Papers ( 150 ) 
MFOl/PCOl Plus Postage . 

♦Audience Response ; Cooperative Learning ; Feedback ; Foreign 
Countries; *Formative Evaluation; Group Activities; Higher 
Education; *Material Development ; *Multimedia Materials ; 
♦Peer Evaluation; Preservice Teacher Education 
University of Lethbridge (Canada) ; Usability 



ABSTRACT 



This paper describes an ongoing research project that uses a 
set of formative evaluation activities to examine a multimedia production 
group project in a teacher education course at the University of Lethbridge 
(Alberta, Canada) . The study presents the notion of peer evaluation and 
target population feedback as tools for scaffolding students' reconstruction 
of multimedia projects. Research will use data collected through students' 
e-mail assignments and projects collected. E-mail submissions will be 
classified into one of the following categories to determine the kind of 
suggestion: graphic design issues; usability/navigation; and functionality. 

In addition, the study will track changes to students' multimedia projects so 
that copies of the projects at various stages of development will be 
retained. Data collection will continue for two or three years or until 
sufficient data has been collected to draw valid conclusions. Contains 10 
references . (MES) 
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Abstract: Multimedia projects are oppoitumties to negotiate meaning through a variety of 
portrayals. However, too often multimedia projects are created with inadequate thought for 
how it will be perceived by others. Multimedia products at times are not user- ftiendly or 
meaningftil because the creator has designed it fiom an egocentric perspective. This paper 
will describe an ongoing research project that uses a set of formative evaluation activities 
involving peer and target population evaluation. These activities are designed to help students 
involved in mul timedia production to develop a greater sense of audience. 



Introduction 

The Faculty of Education at the University of Lethbridge (Canada) has been offering 
a coiurse called Multimedia and Learning as a senior imdergraduate offering for the past three 
years. This course is a study into the tools and educational value of multimedia. The course 
surveys multimedia applications, discusses the val)i| of multimedia for learning, presents 
principles of design and layout, and engages the user in a team approach to development. The 
intent of this course is to provide educators with skills and knowledge to effectively integrate 
and discuss the role of multim edia in learning. It also prepares education students with 
experiences producing multimedia materials so that they are in a better position to use it with 
their own students. An important element of this course is group work. Students in this 
course are grouped into research and production teams. Members of the team are delegated 
topics to research and application tools to become famil iar with, a form of reciprocal learning 
(Brown, Campione, Webber, and McGilly, 1992). This project culminates in the creation of 
a major multimedia product 

Students need to be critical developers as they select tools and representations that 
externalize their thoughts. This takes an imderstanding of the media but more importantly it 
requires a sense of audience. Without a sense of audience, multimedia may fail negotiating 
meanin g. Students will create a more commimicative product if they consider how others 
interact and view their material. To foster this sense of audience, the student teams first 
shares their project with another group through peer evaluations and adjustments are made to 
the product accordingly. The product is then field tested on a small number of individuals 
from the target population. The target populations are individuals who the project was 
designed for outside the multimedia class. Based on that experience further refinements are 
made. Students are then asked to list and are accoimtable for the suggested changes or 
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criticisms of their projects. One of the difficult aspects of teaching a mnltimeHia production 
course is timely and effective feedback at the formative stages of multimedia construction. 
The techniques of peer evaluation and target population testing are powerful incentives to 
make formative changes and refinements 



Background: 

Peer evaluation is the process of students sharing their work with other students who 
have been assigned to assess it. In this particular situation it is for the purposes of formative 
evaluation, to improve the multimedia project that students have created. The idea is that 
others look and interact with a student's work and provide a commentary on the things that 
were good as well as things that could be improved. In reviewing the literature on peer 
evaluation there are two main categories. One area is the use of peer review for the purpose 
of teacher evaluation; the other is peer evaluation in the writing process. Teacher ev^uation 
will not be considered here because this project is more interested in how peer evaluation 
influences learning. There are a fair number of references to peer evaluation into the writing 
process because it is a part of proof reading, having someone else to read written work. The 
value of this activity is rather obvious since the writers of the text knows what they want to 
say they often assume the reader will be able to follow their writing logic. Having someone 
else review writing often will reveal shortcoming in writing process and provide suggestions 
to improve the final product (Benesch, 1985; Bishop, 1986). The same is principle is true 
for multimedia. Students tend to be too close to the work and an external perspective gives 
the developers a sense of audience. Reviewed texts on educational assessment had few 
explicit references to peer evaluation and cases in which there were references to peer 
evaluation, they were imbedded in other approaches or glossed over. This may not be the 
case for all textbooks but clearly this is not a strategy that is a common component of 
educational assessment. It was surprising the difficulty in finding research or publications on 
peer assessment per se. Stiggins (1997) suggests using student as evaluators of skills and 
products. That author further points out that if students are to be used they need to be 
trained. This is also suggested as an important aspect of peer assessment in the writing 
process (Stem, 1992). In fact it is suggested that students come up with the evaluation 
criteria they will use in performance assessment. This is a strategy that is used in a variety of 
domains. At the University of Massachusetts for instance there has been fair amo unt of 
investigation into paired-problem solving which is a form of peer assessment (Whimbley and 
IxKhhead, 1986). It is used in the context of Math as a way of sharing and justifying an 
individual's problems solving strategies with another. The partner acts as a sounding board 
and will use probing of challenging questions to stimulate further thought or nudge thinking 
in a different direction. This approach seemed to have value in the development of effective 
problem solving. 

To tmderstand the value of peer and target population evaluations this p^er draws 
not only on the educational perspective but the multimedia industry as well. Although the 
course is not intended to train people for the multimedia industry, there are some processes 
used by the industry that can be adopted by educators. In the real world when companies 
create quality multimedia products, much planning and testing goes into the process. 
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Effective planning is definitely a necessity, but there is a need for testing as weU and this is 
understood by the industry. Most companies have alpha and beta versions of their software 
that is distributed to potential clients to test. In contrast, education instmctors' on the whole 
have done little with external evaluation in multimedia projects. The reason for this iis 
multifaceted. First, multimedia projects are relatively new to education, most instmctors, 
haven't considered multimedia forms of co mmuni cation to have the same status as linguistic 
forms. Instmctors will often address the issue of verbal and textual forms of commimication 
in their classes and have a set of criteria for evaluating these forms of representation in 
assignments but don't do the same for multimedia. In addition, instmctors may not thinV that 
multimedia deserves the same degree of attention because it is seen as fun rather than a 
serious form of commimication. Another reason may be that the instmctors have had few 
opportunities to commimicate through multimedia themselves so are not sure what makes a 
good project and what does not. This will probably change as we have seen with the 
sophistication of web pages on the Internet. For instance, as people have become more 
discerning they have been drawn by those web pages that are more engaging and logically 
organized. This in turn becomes an expectation for web pages. For instance, there are 
numerous web sites that deal with what a good web site should be like or help designers 
identify common short coming in their design. Another reason is that if the multimedia 
industry fails to produce a quality product the consequences can be rather harsh; the audience 
will ignore their product and use something else, which results in a financial down turn for 
the company. Essentially for multimedia companies, aiidience is everything and companies 
that ignore that don't last long in the competitive maricet place. In education those harsh 
realities are removed so there is less pressure to consider the quality of a multimedia project 
Case in point, provincial or state examinations for most part do not provide options for 
multimedia forms of c ommuni cation. All these are real barriers to using multimedia and in 
particular using peer evaluation as a formative evaluation technique. 

The Study 

The research question is, if students are encouraged to make an accormting of the 
suggestions of others, will that be an impetus for changing their multimedia products? To 
address this question a number of different sources of data will be collected and tabulated. 
During the course on multimedia and learnin g , students report on their peer evaluations and 
the results of their target population field tests through the use of email messages. These 
email messages will be analyzed and categorized. This information will be put into a table, 
which should facilitate drawing inferences and making conclusions. Students' multimedia 
projects will be collected at the various stages of development. Using their projects as 
references one can tell a story of how their projects changed over time. The researcher plans 
to use students’ multimedia projects as exemplars of change. Each group submits a copy of 
their projects before peer and target population evaluations and then again after each form of 
evaluation. 

This study presents the notion of peer evaluation and target population feedback as 
tools for scaffolding students’ reconstruction of multimedia projects. This is a work in 
progress. It is hoped that researchers and instructors will be able to use the principles coming 
out of this research and apply those to their own multimedia productions or related courses. 
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The results of this research address the value of audience in changing student multimedia 
constructions and suggest implications for other courses. 



Data and analysis: 

Ongoing data collection will begin in January 2000. This will continue for two or three 
years or until sufficient data has been collected to draw valid conclusions. There are two 
kinds of data that the study will use, data collected through email assignments fix>m offerings 
of the course and projects collected. Students’ email messages wUl be categorized and 
tabulated (as seen Figure 1). The contents of student email submissions will be classified into 
one of the following categories to determine the kind of suggestions made: 

• Graphic Design issues: This category will cover issues like style, color, placement of 
images, text, font size, and the over look and feel of the project. 

• Usability/navigation: This category covers issues of interactivity, the degree to which 
it is user fiiendly. This will be reflected in comments, relating to how the user 
navigates through the project or how the project is structured. For instance, is it clear 
how to get firom one part to the next, can users get to parts of the p’oject they want to, 
is there a need for further links, etc. 

• Functionality: Fimctionality will deal with more fundamental questions. Does it 
accomplish its piirpose? Is it worthwhile? 

It should be noted that the evaluators are not given any explicit criteria or categories to 
evaluate products, although these categories are covered in class in regards to developing 
good multimedia. Evaluators are asked to identify those aspects of the project that were good 
and those things that might be improved upon. 

Once a comment has been categorized it will be determined if it is a positive comment or 
a suggestion for change. As part of an assignment for the course, students will be held 
accountable for each of the suggested changes made by peers or the target population. They 
will either be expected to reflect that change in their woik or justify why they did not. A 
further category will be the justifiable nonchanges. These would be changes that are 
identified by the project group as being worthy to pursue but for one reason or another (time, 
resoiirces, expertise) are not able to inq)lement This does not suggest students will be 
compelled to make changes. Students do not need to make changes to get credit for 
assignments. However, they are expected to address the suggestions either by indicating the 
suggestions are not necessary, the change is a justifiable nonchange, or they see the value of 
the change and make the change accordingly. Tabulating this information will provide data 
on the usefulness of peer evaluation for changing the multimedia projects. As an after 
thought there may be an alternative to email, that is online databases. This too mi gh t 
facilitate summarizing the data. 
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In addition this study will track changes to students multimedia projects so copies of 
students’ projects at various stages of development will be retained. These juxtaposed will 
the tabulated data will help teU the story of how change takes place. The study will use these 
projects as exemplars in presentations or as screen displays in reports. Table 1 illustrates a 
tentative form that will be used to collect and tabulate the data that will be collected and 
categorized. 
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Role of audience in changes to usability 
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Table 1: This is a sample data collection form which vidll be Used to tabulate results from 
student peer and target population evaluations on the category of usability. Each 
category would a have similar table associated with it. 



There are two ways students may develop a greater sense of audience. One is that 
they see the value of having others review their work. This might be measured to the degree 
they think valuable information will come form peer and target population involvement in 
the production process. Another measure of a sense of audience is, the extent student 
creators are able to take the perspective of others in the production process. This is a difficult 
factor to collect information on. One way address this issue is to determine to what extent 
peer and target population comments are useful in future project development? Does the 
experience actually raise an awareness of a sense of audience for participating students? This 
will be difficult to docrunent or track over the long term. The nature of these suggestions 
may be endemic to a particular project, so to have students demonstrate what they have 
learned through the development of a different project may not be valid. However, perhaps 
one can infer this kind of transfer if students are able to identify the principles that they have 
learned fix>m their peer evaluations. So an additional question will be posed to students for 
them to identify the principles they have learned from the peer and target population 
evaluations. It is hoped that through the course content student will have the language to 
articulate these principles. Another question wiQ be asked at the end of the session to see to 
what extent they feel target and peer evaluation is a part of the evaluation process. 



Findings 

The information from Table 1 will be used to identify patterns in the data. For 
instance, the measure of student sense for audience comes fix>m either a low number of 
changes being suggested or a high percentage of the suggested changes being acted on. The 
munber of changes over suggestions should be high, if students have a good sense of 
audience with justified nonchanges taken into account. In other words, groups that have been 
told there are problems with their projects should make changes to accommodate those 
suggestions. However, there will be expected disagreements fix>m time to time in terms of 
suggestions made and the degree to which the creators of the project sense it is an issue. 
There are two possible explanations, either the creator group ignores the perspective of the 
audience or the evaluator group are making unreasonable suggestions. Other patterns may 
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emerge from this table as well. For instance the number of positive comments mi ght be 
viewed as an overall evaluation of the project. Looking at the differences in the number and 
type of suggestions between the peer evaluation and the target population would also be 
interesting information to collect. Identifying further changes through self-evaluation might 
further provide evidence of a heightening sense of audience. There is no need for a control 
group because each project is a control in the sense that before the target population and peer 
evaluation, the project is meant to be in a finished state. So the control is the state of the 
project before peer and target evaluation have been accomplished. The numbers that will be 
collected are not sufficient to do any kind of inferential statistics so this becomes a 
qualitative study 

There are no definitive results since this a study in progress. However, data from 
previous years were put into the table to test the categorization scheme. The pilot study data 
was obtained from a manually maintain ed email archive. Although the data was sketchy and 
was not categorized as finely as this proposal suggests, patterns began to emerge. One 
pattern was that through peer evaluation and target evaluation, numerous suggestions for 
change were made in all categories but functionality. Another result was that the production 
group acted on most suggestions. One might argue that these changes were stimulated not by 
peer evaluation but by the nature of the assignment. Controlling for that would be difficult 
but the course assignments could be setup so that they don't get any credit for addressing the 
suggested changes. However this in turn might skew the degree to which the students look 
seriously at suggested changes. Giving students credit for making changes is thought to 
stimulate them to be more sensitive to the issue of audience. Suggestions not acted on may 
have fallen into different categories. First the development group may not have shared the 
value of these suggestions. Alternatively, the suggestions may have fallen into the justified 
nonchange which would suggest that the development group realized the value of the 
comment but did not have resources or time to act on it. It was interesting to note that were 
only one or two suggestions in terms of functionality. However, through imderstanding the 
context this result becomes explainable. Suggestions regarding functionality address the very 
intent of the project. Fimctionality addresses the project's purpose, was it successful 
achieving its purpose or was the purpose useful? The peer or target evaluators, knowing that 
by this time the production team had gone to great lengths to create the product to date may 
have suppressed these issues. Instructional interventions are now being considered to 
accommodate for this. For instance, if students went through a peer review early in the 
process before production commenced as a sort of proof of concept, it would be interesting to 
see the number of comments that emerge regarding the functionality of the project Another 
explanation is that the audience viewed the projects as being functional. A more systematic 
data collection process should provide a better indication of how student projects change and 
why. At this point it may be premature but the pilot study suggests that through greater 
audience appreciation (peer and target evaluation) many useful changing can be made to 
multimedia projects. 



Conclusions 

Designing and developing multimedia projects may involve different skills and engage different 
learning styles than that used in the writing process. However, because both processes involve creating a 



product that will negotiate meaning and facilitate communication with others, both ^proaches can benefit from 
a sense of audience. In this case the role of audience comes through student assignments that involve having 
multimedia projects reviewed by peers as well as individuals fi-om the target population. This study reported an 
ongoing project that documents the role of audience on changes to students' multimedia projects. It is 
speculated that feedback fiom an audience is a powerfiil influence on implementing changes to multimedia 
projects. On a broader note it is thought that the audience's perspective differs from the creators perspective and 
this information can help make multimedia projects more useful. With the current emphasis in the curriculum 
on students creating alternative ways of presenting and portraying ideas, students need a better sense that 
meaning is a negotiated issue. Using techniques like peer and target evaluations to improve multimedia projects 
will highlight the importance of this. 
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